Image processing technologies to process digital image data have been widely known. As one of those image processing technologies, a blur restoration technology has been known. The blur restoration technology is a technology to restore blurred images.
The blur restoration technologies include, for example, a noise removal technology (denoise), a haze removal technology (dehaze), and a super resolution technology (super-resolution) (for example, refer to NPL 1). The super resolution technology will be described below as an example of the blur restoration technologies.
The super resolution technology is an image processing technology to raise the resolution of image data. The super resolution technologies include, for example, the following two technologies.
The first super resolution technology is a multiple-frame super resolution technology. The multiple-frame super resolution technology is a technology to generate a piece of high resolution image data by using a plurality of pieces of image data (a plurality of frames) that composes a motion video or are generated by consecutive shooting (for example, refer to PLT 1). As described above, the multiple-frame super resolution technology requires a plurality of pieces of image data to achieve high resolution. Thus, the multiple-frame super resolution technology is incapable of generating a piece of high resolution image data from a piece of image data.
The second super resolution technology is a learning based super resolution technology. The learning based super resolution technology is a technology to create a dictionary based on learning processing in advance and raise the resolution of a piece of image data by using the dictionary (for example, refer to PLTs 2 and 3). Since the learning based super resolution technology uses a dictionary, the learning based super resolution technology is capable of achieving a higher super resolution than the multiple-frame super resolution technology that uses a smaller number of pieces of referenced image data.
The learning based super resolution technology will be further described with reference to the drawings. The learning based super resolution technology includes “a learning phase” and “a super resolution phase” in general. “The learning phase” is a phase in which a dictionary that is used for super resolution processing is created. “The super resolution phase” is a phase in which a high resolution image is generated from a low resolution image by using the dictionary.
In the learning based super resolution technology, a device may carry out both phases. Alternatively, a plurality of devices may carry out the respective phases individually.
To make the description clearer, description using devices for the respective phases will be made below.
FIG. 10 is a diagram illustrating an example of a configuration of a super resolution system 900 that is related to the present invention.
The super resolution system 900 includes a dictionary creation device 910, a dictionary 920, and a super resolution image generation device 930.
The dictionary creation device 910 carries out a learning phase. Specifically, the dictionary creation device 910 creates patches (patch pairs 531), which are used in a super resolution phase, based on learning images 51, and stores the created patch pairs 531 in the dictionary 920.
The dictionary 920 stores the patch pairs 531 which the dictionary creation device 910 creates for the creation of a super resolution image.
The super resolution image generation device 930 carries out the super resolution phase. Specifically, the super resolution image generation device 930 generates a restored image 55 (a high resolution image) by using an input image 54 (a low resolution image) and the patch pairs 531, which are stored in the dictionary 920.
The respective phases will be further described.
FIG. 11 is a diagram for a description of the learning phase. Processing in the learning phase will be described by using FIGS. 10 and 11 in combination.
The dictionary creation device 910 receives high resolution images for learning (the learning images 51). The dictionary creation device 910 generates low resolution images (blurred images 52) by lowering the resolution of the learning images 51.
The dictionary creation device 910 cuts out image portions within predetermined ranges (high resolution patches 511) from the learning images 51. Further, the dictionary creation device 910 cuts out image portions (low resolution patches 521), that correspond to the cut-out high resolution patches 511, from the blurred images 52.
The dictionary creation device 910 generates patch pairs 531 by combining the high resolution patches 511 with the low resolution patches 521. The dictionary creation device 910 stores the patch pairs 531 in the dictionary 920.
FIG. 12 is a diagram for a description of the super resolution phase.
The super resolution image generation device 930 receives the input image 54.
Based on the input image 54, the super resolution image generation device 930 generates patches (input patches 541) to be compared with the low resolution patches 521 in the patch pairs 531.
Based on the generated input patches 541, the super resolution image generation device 930 selects patch pairs 531 by referring to the dictionary 920. More specifically, the super resolution image generation device 930 operates, for example, in the following manner.
The super resolution image generation device 930 calculates similarities between the input patch 541 and the low resolution patches 521 in all patch pairs 531. Based on the similarities, the super resolution image generation device 930 selects a patch pair 531 that includes the most similar low resolution patch 521. The high resolution patch 511 of the selected patch pair 531 becomes a patch (a restoration patch 551) that is used for compositing.
The super resolution image generation device 930 selects patch pairs 531 that correspond to all input patches 541. By using high resolution patches 511 in the selected patch pairs 531 as restoration patches 551, the super resolution image generation device 930 generates a restored image 55 (a super resolution image).